Title: Backups, Options, and Effort
Date: 2016-05-06 13:15
Category: backups
Tags: backups, CLI, rsnapshot, rsync, network, obnam, ncdu, gpg, luks, dmcrypt, awk

**Followup [Backups Revisited] 10FEB2017**
**Followup [Display]**


###Backups
blah blah blah.
We all know, or should know, that backups are important.


###Options
There are a [Lot of Options] out there.
A lot of them don't do what I want.
There are a few criteria that I desire, deltas, encrypted transfer, encrypted storage, space conservation, remote backup.
There are few programs that did these, and did them in a way I liked.
[rsnapshot] and [obnam] are the two I put most of my attention into.
I did make an exception for `rsnapshot` in that it does not do encryption itself, but luks+dmcrypt deals with that.


####rsnapshot
I started off playing with `rsnapshot` as it sounded very tasty, with its [rsync] use, and log like rotation of backups, and hard linking files between backups.
To start with, I found [pigmonkey's post] about their use of `rsnapshot` (I've been subscribed to their blog for years, and hey! A post about what I'm trying to use!).
Their script ([cryptshot]) worked great locally, though the first thing I did was to tweak it to gpg decrypt my luks keyfile.

    :::bash
    gpg2 --no-permission-warning --no-tty -qd $KEYFILE | cryptsetup luksOpen --key-file=- $volume $name

<sup>NOTE: --no-permission-warning is there due to the use of `sudo -E`.</sup>

This was all well and good, but I wanted backups to happen automatically.
Since I use a laptop, hooking in the external drive would be a thing that prevents this from being automagic.
So, I start tweaking the script, and make a 'backups' user on my pi, giving them `NOPASSWD` permissions in `/etc/sudoers` with exact augments provided, so that the account can't be missused.
I figure out that I can stream the gpg decrypted luks key via ssh as I only want it to live on my laptop, and have the process controlled by ssh-agent and gpg-agent being cached.

    :::bash
    gpg2 --no-permission-warning --no-tty -qd $KEYFILE | ssh $re_addr "sudo cryptsetup luksOpen --key-file=- $volume $name"

Then I actually started playing with `/etc/sudoers.d` as it has some benefits, and adjusting permissions.
I'm going to get to this section later, as I went through a number of revisions.


At this point I made a user on the backup server, and locked them down.
I'll go into specifics later.

So, this where I ran into an annoying wall.
`rsnapshot` only works on a 'pull' model.
This is fine if everything is local.
And also means you can pull an external server's content.
But not the other way around...

Which is exactly what I wanted to do, as my GPG key is staying on my computer and is only unlocked for a few hours.
I periodically reopen the key when I refresh mutt/pianobar, which would mean if-i-am-around -> do-the-backup.

So I set my `rsnapshot` plans down, and look at my options again.


####obnam
For the next two days I poked `obnam` instead.
It seems nice, works on the push philosophy, does deltas, can use gpg encryption.

While trying it out I run into a few issues though.
`obnam` uses `gpg1`, or at the very least `/usr/bin/gpg`.
A future (soon) version of debian sid may change this, as I hear news of gpg2 being `/usr/bin/gpg` in the works. <sup>(has happened now, on debian sid 17AUG2016)</sup>

Anyway, this is an 'issue', as my low-importance key (for things like `pass`, local files, etc) is a gpg2 key, and gpg1 just doesn't see it.

\**sigh*\*

Next up: It takes longer to backup than `rsnapshot`.
I didn't do scientific testing here, but it really seemed to take 10-15 minutes to do a delta backup with `obnam`, where `rsnapshot` knocked it out in about 5-7 minutes.
(context: local backups with external plugged straight into laptop, total hard drive usage <300gb out of 500gb, external drive is 1tb)

The gpg issue isn't `obnam`'s fault, and the backup speed is just a little annoying.

The third issue was a deal breaker...
`obnam` is tar based, and offers two ways of accessing the backups, fuse and extraction.
The added layer of fuse makes unfortunately ***really*** slows down your interaction speed.
If I run [ncdu] on my laptop, it takes ~5 minutes to the scan.
If I have one backup via `rsnapshot` it takes ~5 minutes to scan (or if I just point `ncdu` at one hourly backup).
If I have one backup in `obnam` it takes >3 hours to `ncdu`... ***ffs!***

So, back to the drawing board.


####Back to rsnapshot
At this point I look at `rsnapshot` again.
I mostly put it down, as thinking about the threat model of letting the backup server have access to my laptop was going to take effort.

I decide I will have to give the backup user limited access to my laptop, if I want backups to work and be automatic.
There will be a backup user on both the pi and the laptop, passwords will be disabled with `passwd -l` which still allows key based ssh login.
The ssh key will have restricted IPs, and point at a script that checks the incoming commands (even though sudo checks too).


###End Setup
Here is a summary of how I tied everything together.


####Users
Both the laptop and the server have a backup user, with their password disabled

    :::bash
    # Add the user
    sudo adduser backups
    # Become the user
    sudo su - backups
    # I'm lazy, this makes ~/.ssh with the right premissions.
    ssh localhost
    exit
    # make a non standard key
    ssh-keygen -b BIG_NUMBER_HERE
    exit
    # Disable the user password
    sudo passwd -l backups

<sup>NOTE: Non standard key sizes (and primes) are a good thing!<sup>


####SSH
Installed to `/usr/local/sbin/val-back-cmd.sh` for permission security.  

These accounts have an `authorized_keys` file.
The keys are restricted to set IPs, and run a script that checks the incoming commands.

    :::bash
    #~/.ssh/authorized_keys
    from="10.0.0.2,10.0.0.3",command="/usr/local/sbin/val-back-cmd.sh" SSH_KEY_HERE

Paired with:

    :::bash
    #! /bin/bash

    laptop_host="moving-computer-of-doom"
    remote_host="tiny-server-of-doom"

    if [ $HOSTNAME = $laptop_host ]; then
        case "$SSH_ORIGINAL_COMMAND" in
            *\&*|*\|*|*\;*|*\>*|*\<*|*\!*)
            exit 1
            ;;

            ## This needs to be set in /etc/rsnapshot.conf rsync_long_args
            /usr/bin/rsync\ --server\ --sender*)
            sudo $SSH_ORIGINAL_COMMAND
            ;;

            *)
            exit 1
            ;;
        esac
    elif [ $HOSTNAME = $remote_host ]; then
        case "$SSH_ORIGINAL_COMMAND" in
            *\&*|*\|*|*\;*|*\>*|*\<*|*\!*)
            exit 1
            ;;

            mount*)
            sudo $SSH_ORIGINAL_COMMAND
            ;;

            umount*)
            sudo $SSH_ORIGINAL_COMMAND
            ;;

            cryptsetup\ luksOpen*)
            sudo $SSH_ORIGINAL_COMMAND
            ;;

            cryptsetup\ luksClose*)
            sudo $SSH_ORIGINAL_COMMAND

            rmdir*)
            sudo $SSH_ORIGINAL_COMMAND
            ;;

            mkdir*)
            sudo $SSH_ORIGINAL_COMMAND
            ;;

            rsnapshot*)
            sudo $SSH_ORIGINAL_COMMAND
            ;;

            *)
            exit 1
            ;;
        esac
    fi

It is worth noting that you do need to manually connect from laptop-to-server, and server-to-laptop at least once, so that the host key is accepted.

[Example val-back-cmd.sh]


####Sudoers
Eariler I talked about setting up `/etc/sudoers`.

But thinking about it, lets set this up better, and use `/etc/sudoers.d/`.
Now we don't have to worry about the order, and not backups will not be effected by future `sudo` package updates.
To make a new entry, `sudo visudo -f /etc/sudoers.d/backup_sudo`.

    :::bash
    Cmnd_Alias MOUNT = /bin/mount /dev/mapper/crypt-YOUR_UUID_HERE /mnt/YOUR_UUID_HERE, /bin/umount /mnt/YOUR_UUID_HERE

    Cmnd_Alias LUKS = /sbin/cryptsetup luksOpen --key-file\=- /dev/disk/by-uuid/YOUR_UUID_HERE crypt-YOUR_UUID_HERE, /sbin/cryptsetup luksClose crypt-YOUR_UUID_HERE

    Cmnd_Alias DIR = /bin/rmdir /mnt/YOUR_UUID_HERE, /bin/mkdir /mnt/YOUR_UUID_HERE

    Cmnd_Alias RSYNC = /usr/bin/rsync

    Cmnd_Alias RSNAPSHOT = /usr/bin/rsnapshot hourly, /usr/bin/rsnapshot daily, /usr/bin/rsnapshot weekly, /usr/bin/rsnapshot monthly, /usr/bin/rsnapshot yearly

    Cmnd_Alias CRYPTSHOTR = /usr/local/sbin/cryptshotr -q hourly, /usr/local/sbin/cryptshotr -q daily, /usr/local/sbin/cryptshotr -q weekly, /usr/local/sbin/cryptshotr -q monthly, /usr/local/sbin/cryptshotr -q yearly

    ## On laptop, when remote is accessing
    ## Laptop needs RSYNC and CRYPTSHOTR, as 'sudo cryptshotr' will handle the decrypting/mounting/mkdir. laptop backup user needs RSYNC. 'cryptshotr-cron' needs CRYPTSHOTR.
    ## Remote server needs MOUNT, LUKS, DIR, RSYNC, and RSNAPSHOT.
    %backups ALL=(ALL) NOPASSWD: RSYNC, CRYPTSHOTR

With the commands broken up by type, it makes it much more readable, and I don't need to give the backup server CRYPTSHOTR access, etc.

[Example backup_sudoers]


####rsnapshot
Installed to `/etc/rsnapshot.conf`  

***TABS***, you must use tabs.
Really, options and augments have to be tab separated.
I've warned you.

There are three big settings, as well as the `interval` and `exclude` section.
The follow is just to highlight those settings, it is incomplete.

    :::bash
    snapshot_root   /mnt/YOUR_UUID_HERE

    #...

    ### Intervals ### {{{
    ## Must be unique and in ascending order
    retain          hourly          24
    retain          daily           7
    retain          weekly          4
    retain          monthly         12
    retain          yearly          5
    ### End Intercals ### }}}

    #...

    rsync_long_args         -ev --rsync-path=/usr/bin/rsync

    #...

    ## For local cryptshotr
    backup  /               doom/
    ## For remote cryptshotr
    #backup backups@moving-computer-of-doom.local:/          doom/

[Example rsnapshot]


####cryptshotr
Installed to `/usr/local/sbin/cryptshotr` for permission security.  

For `cryptshotr`, my modification of `cryptshot` by pigmonkey, I ended up doing a few things.
I already talked about using a GPG encrypted LUKs keyfile.
The rest of my changes are going to be summarized here.


* ☑ Uses GPG with LUKs key, so it is secure at rest.
    * GPG keys are cached with gpg-agent, which I periodically refresh for mutt/pianobar.
* ☑ Can backup over the network.
    * Checks if remote feature is enabled in settings.
    * Checks if connected to home network, via MAC address.
    * Checks if remote server is pingable.
    * Checks if remote volume is connected.
    * Will then pipe gpg decrypted LUKs key over ssh, to open volume; then mount container.
    * This works via my Pi at home
* ☑ Added a 'quiet' mode for less crontab emails
    * I expect a number of backups to fail:
        * My laptop is moble
        * My gpg-agent will time out
        * My server might not recover from the router rebooting <sub>(working on that)</sub>

[cryptshotr code]


####cryptshotr-cron
Installed to `/usr/local/sbin/cryptshotr-cron` for permission security.  

I wrote a wrapper to deal with schduling run times, and running periodicities in order.
`crontab` would have ran multiple periodicities in parallel if they were scheduled at the same time.
This would cause all besides the first to fail, as the container would already be open.
While I considered teaching the script to not care, I decided that leaving it as is would be more tamper-evident.


* ☑ log file
    * Will log periodicities' pass/fail
    * On pass, will purge previous entries for periodicity
* ☑ Will check periodicity's last run epoch against current time
    * If time not met, skips periodicity.
    * If missing runs periodicity.
    * If last run failed, run.

[cryptshotr-cron code]


####crontab
To finish it all up, I call cryptshotr-cron in crontab every half hour.

    :::php
    */30 * * * * /usr/local/sbin/cryptshotr-cron


###My git repo
I have set up a **[git repo]** with the files in one place, if people wish to use my work (built off of pigmonkey's).


###TODO
* Make script for server to detect orphaned volume mount (backup failed/hung/laptop closed), and shut the container.


###Links
[Backups Revisited]  
[Display]  
[Lot of Options]  
[rsnapshot]  
[obnam]  
[rsync]  
[pigmonkey's post]  
[cryptshot]  
[ncdu]  
[Example rsnapshot]  
[cryptshotr code]  
[cryptshotr-cron code]  
[git repo]  


[Backups Revisited]: {filename}014-backups_revisited.md
[Display]: {filename}008-backups_conky_display.md
[Lot of Options]: https://wiki.archlinux.org/index.php/Synchronization_and_backup_programs
[rsnapshot]: http://rsnapshot.org/
[obnam]: http://obnam.org/
[rsync]: https://rsync.samba.org/
[pigmonkey's post]: https://pig-monkey.com/2012/09/cryptshot-automated-encrypted-backups-rsnapshot/
[cryptshot]: https://github.com/pigmonkey/backups/blob/master/cryptshot.sh
[ncdu]: https://dev.yorhel.nl/ncdu
[Example val-back-cmd.sh]: https://notabug.org/demure/cryptshotr/src/master/val-back-cmd.sh
[Example backup_sudoers]: https://notabug.org/demure/cryptshotr/src/master/backup_sudoers
[Example rsnapshot]: https://notabug.org/demure/cryptshotr/src/master/rsnapshot.conf
[cryptshotr code]: https://notabug.org/demure/cryptshotr/src/master/cryptshotr
[cryptshotr-cron code]: https://notabug.org/demure/cryptshotr/src/master/cryptshotr-cron
[git repo]: https://notabug.org/demure/cryptshotr

<cr><sub>**Edited 18MAY2016**: Improved val script's cryptsetup slightly, suggested by pigmonkey.</sub>
